Public Speaking Trainer With 3-D Simulation and Real-Time Feedback

ABSTRACT

A public speaking trainer has a computer system including a display monitor. A microphone is coupled to the computer system. A video capture device is coupled to the computer system. A biometric device is coupled to the computer system. A simulated environment including a simulated audience member is rendered on the display monitor using the computer system. A presentation is recorded onto the computer system using the microphone and video capture device. A first feature of the presentation is extracted based on data from the microphone and video capture device while recording the presentation. A metric is calculated based on the first feature. The simulated audience member is animated in response to a change in the metric. A score is generated based on the metric. The score is displayed on the display monitor of the computer system after recording the presentation. A training video is suggested based on the score.

CLAIM OF DOMESTIC PRIORITY

The present application claims the benefit of U.S. ProvisionalApplication No. 62/036,939, filed Aug. 13, 2014, which application isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates in general to learning game systems and,more particularly, to a three dimensional (3-D) simulator with real-timeanalysis, feedback, and remediation for public speaking practice andimprovement.

BACKGROUND OF THE INVENTION

Public speaking is a common activity in today's society. Beginning ingrade school classrooms, where students are called upon to answerquestions or must give reports in front of the class, through careers aslawyers, politicians, teachers, store managers, and many otheroccupations. At public and private meetings across the globe, peoplestand before crowds to deliver committee reports, financial reports, ortechnical presentations, answer questions, announce news, or otherwisereport information to a crowd. FIG. 1 illustrates speaker 10 performinga presentation in front of audience members 12, 14, and 16. In FIG. 1,presenter 10 is speaking at a board meeting.

Public speaking is a challenging skill that almost everyone couldimprove upon, and very few feel totally comfortable performing. Somepeople fear public speaking to the point of physical distress, nausea,and feelings of panic. A number of methods have been proposed toovercome fear of public speaking, or to improve public speaking skills.Many attempt to rehearse public speaking. Practice can be performed infront of a mirror, or in front of a small group of friends and family.Those methods of practicing offer some benefit, but the feedbackreceived by the presenter is minimal. A mirror offers almost nothing.Friends and family may applaud or give an encouraging comment, but areunlikely to offer serious constructive feedback.

A person can practice a speech while filming the presentation orrecording the audio, and then review the recording to determine whereimprovements could be made. However, without a professional to reviewthe performance, only major issues are likely to be noticed and improvedupon. Reviewing a recording takes significant time when only a fewspecific points of the practice speech contain issues worth noting forimprovement. In addition, feedback is not generally instantaneous, oreven quick. A person may not review the tapes until a significantlylater time. If the person wants a skilled second party to review thetapes, days or weeks could pass before feedback is received.

The challenges with prior art methods for public speaking practice meanthat practice is time consuming and of limited value. The result is thata speaker may practice a speech once or twice prior to public speaking,but will not continue with practice to develop and perfect publicspeaking skills. Any benefit from practicing a speech once or twice islost because the speaker does not continue the practice to reinforcepublic speaking skills.

In some work environments, where public speaking is a critical skill foremployees, experts are hired to come into the company and trainemployees on public speaking. The experts are expensive and, once thetraining is complete, do not stick around to promote retention andreinforcement of skills. Moreover, speaking before an expert is asignificantly different experience than speaking before a crowd. Anideal public speaking training solution would simulate the feeling ofspeaking before a crowd similar to the crowd that a student or employeewould be speaking in front of in real life.

Current presentation and public speaking training solutions do not offersimulations and real-time feedback, limiting user engagement.Furthermore, current solutions lack sufficient mechanisms for practice,assessment, and reinforcement resulting in poor training continuity,sub-optimal retention, and loss of skills. Current training solutionsproduce poor returns on training investment.

SUMMARY OF THE INVENTION

A need exists for an enhanced presentation and public speaking trainingsystem with environmental simulation and real-time feedback.Accordingly, in one embodiment, the present invention is a method ofpublic speaking training comprising the steps of providing a computersystem including a display monitor, providing a microphone coupled tothe computer system, providing a video capture device coupled to thecomputer system, rendering a simulated environment including a simulatedaudience member on the display monitor using the computer system,recording a presentation onto the computer system using the microphoneand video capture device, extracting a first feature of the presentationbased on data from the microphone and video capture device whilerecording the presentation, calculating a metric based on the firstfeature, animating the simulated audience member in response to a changein the metric, generating a score based on the metric, and displayingthe score on the display monitor of the computer system after recordingthe presentation.

In another embodiment, the present invention is a method of publicspeaking training comprising the steps of providing a computer systemincluding a display monitor, providing a microphone coupled to thecomputer system, providing a video capture device coupled to thecomputer system, rendering a simulated audience member on the displaymonitor using the computer system, recording a presentation onto thecomputer system using the microphone and video capture device,extracting a feature of the presentation based on a data from themicrophone and video capture device while recording the presentation,and animating the simulated audience member in response to the feature.

In another embodiment, the present invention is a method of publicspeaking training comprising the steps of providing a computer systemincluding a display monitor, providing a microphone coupled to thecomputer system, rendering a simulated audience member on the displaymonitor using the computer system, recording a presentation onto thecomputer system using the microphone, and animating the simulatedaudience member based on a data from the microphone.

In another embodiment, the present invention is a method of publicspeaking training comprising the steps of providing a simulated audiencemember, receiving a presentation, and animating the simulated audiencemember in response to the presentation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a person presenting at a board meeting;

FIG. 2 illustrates a computer system to simulate environments andprovide real-time feedback for a public speaking training system;

FIG. 3 illustrates an electronic communication network used by thecomputer system to communicate with a computer server;

FIG. 4 illustrates a cloud network;

FIG. 5 illustrates a computer application utilizing a plurality ofsoftware engines;

FIG. 6 illustrates a software application programming interface (API);

FIG. 7 illustrates physical and other presentation inputs to a speechanalysis engine;

FIG. 8 illustrates a supervised machine learning classificationalgorithm;

FIG. 9 illustrates initial setup of the public speaking training system;

FIG. 10 illustrates an activity selection screen of the public speakingtraining system;

FIG. 11 illustrates a presentation being analyzed to provide real-timefeedback and scores and ratings for the presentation;

FIGS. 12 a-12 e illustrate a person practicing a presentation with thepublic speaking training system;

FIG. 13 illustrates a summary screen of the public speaking trainingsystem after completion of a presentation;

FIG. 14 illustrates a user replaying a performance to review noteworthyaspects; and

FIGS. 15 a-15 b illustrate a supervisor reviewing the performance ofseveral employees.

DETAILED DESCRIPTION OF THE DRAWINGS

The present invention is described in one or more embodiments in thefollowing description with reference to the figures, in which likenumerals represent the same or similar elements. While the invention isdescribed in terms of the best mode for achieving objectives of theinvention, those skilled in the art will appreciate that the disclosureis intended to cover alternatives, modifications, and equivalents as maybe included within the spirit and scope of the invention as defined bythe appended claims and claims equivalents as supported by the followingdisclosure and drawings.

A presentation training system including a simulated presentationenvironment and real-time feedback is presented. A user 10 uses thepresentation training system to practice presentations and developpublic speaking skills. The training system simulates an environmentsimilar to the environment that user 10 will actually be presenting in,and a crowd similar to the crowd that the user will be speaking to. Thetraining system observes user 10 presenting via optical, audio, andbiometric inputs, and is able to give real-time feedback during thepresentation, summarize performance problems after the performance, andprovide tips and tutorials for improvement. The training system providesa dynamic, goal based, educational experience.

FIG. 2 illustrates a computer system 20 used to execute program code forthe public speaking training and simulation software. Computer system 20is a computer system located near user 10 while the user practices apresentation. Computer system 20 can be a desktop or laptop personalcomputer (PC), a video game console, a mobile phone or tablet, a virtualreality headset, or other electronic device able to execute softwarecode to observe a presentation and present feedback to user 10. Computersystem 20 includes a central processing unit (CPU) 22, mass storage orhard drive 24 coupled to be read from and written to by the CPU, randomaccess memory (RAM) 26 coupled to be used by the CPU as temporary datastorage, input peripherals 28 used to capture a performance of the user,communication port 30 used to communicate over a network usingcommunication link 32, and a graphics display or monitor 34.

Program code for the training system is distributed via a portable massstorage medium, such as a compact disk (CD), digital versatile disk(DVD), or thumb drive. The program may also be downloaded over theinternet or another network. Program code for the training system isinitially stored in mass storage 24. The program code may be downloadedover the internet directly to mass storage 24, or the program code isinstalled from a CD or DVD onto mass storage 24. In some embodiments,the program code runs from the CD or DVD rather than being installedonto mass storage 24.

When user 10 wants to train with the presentation training system, theuser uses an interface of computer system 20 to begin execution of theprogram code. CPU 22 is instructed by an operating system (OS) ofcomputer system 20 to copy the program code from mass storage 24 intoRAM 26. In embodiments where the program code is not installed onto massstorage 24, the program code is copied from a CD, DVD, or other portablestorage media into RAM 26. CPU 22 then begins reading and executing theinstructions embodied in the program code now stored in RAM 26. CPU 22also uses RAM 26 to store temporary data such as recordings of theperformance of user 10, features extracted from the presentation, andcalculated ratings and statistics on the presentation. Such data may bestored permanently by copying the data to mass storage 24 or sending toa server via communication port 30.

CPU 22 receives streaming data representing the presentation of user 10via input peripherals 28. Input peripherals 28 include a microphone, acamera, a keyboard and mouse, biometric inputs, and other peripheralscapable of conveying information regarding a presentation of user 10.Data from input peripherals 28 streams to CPU 22, which handles the dataas instructed by the program code of the training system. CPU 22 appliesanalysis algorithms to the data from input peripherals 28 to extractpresentation features, calculate scores and ratings, and generate otherfeedback to benefit user 10. In some embodiments, CPU 22 sends selectstreaming data from input peripherals 28 to other computers, such as alocal server or cloud server, for analysis via communication port 30. Inother embodiments, CPU 22 sends the completed presentation by user 10 toa server via communication port 30 for remote storage, or data regardingthe completed presentation for review by a supervisor.

Communication port 30 represents any hardware device of computer system20 capable of communicating with other computer systems. In oneembodiment, communication port 30 is a wired or wireless Ethernetadapter. Communication link 32 represents a network cable or a wirelesslink to another computer system or to a local network router.

Display 34 shows a graphical user interface with contents controlled byCPU 22 executing the program code of the training software. While user10 is giving a presentation to the training system, display 34 shows asimulated environment including a realistic room with a crowd of people.The crowd reacts realistically, using gestures, movement, posture, andfacial expressions, based upon the content of the presentation and howwell the speech is delivered by user 10. Feedback to help user 10develop presentation skills is presented visually on display 34 duringthe speech, and a summary including final performance ratings and tipsis shown on the display after a presentation is complete. Display 34 mayshow user 10 giving the speech as a thumbnail while the user ispresenting, and also shows the video of the speech after thepresentation is complete for review by the user.

Display 34 is integrated into computer system 20 in some embodiments,such as when computer system 20 is a cell phone, tablet, or virtualreality headset. In other embodiments, display 34 is an external monitorconnected to computer system 20 via a video cable.

FIG. 3 illustrates an electronic communication network 52 that computersystem 20 connects to via communication link 32. Electroniccommunication network 52 represents any digital network, such as theinternet, a private wide-area network (WAN), a corporate network, or ahome local area network (LAN). Electronic communication network 52includes a plurality of network cables, switches, routers, modems, andother computer systems as necessary to route data traffic betweencomputer system 20 and other computer systems connected to theelectronic communication network.

Computer system 20 is located at a home, office, or other locationaccessible by user 10. Computer system 20 communicates with computerserver 54 via electronic communication network 52. Data packetsgenerated by computer system 20 are output through communication link32. Electronic communication network 52 routes the data packets from thelocation of computer system 20 to the location of computer server 54.Finally, the packets travel over communication link 56 to computerserver 54. Computer server 54 performs any processing necessary on thedata, and returns a message to computer system 20 via a data packettransmitted through communication link 56, electronic communicationnetwork 52, and communication link 32. Computer server 54 also storesthe data received from computer system 20 to a database or other storagein some embodiments.

Cell phone 66 is connected to electronic communication network 52 viacommunication link 68, and tablet computer 70 is connected to theelectronic communication network via communication link 72.Communication links 68 and 72 can be cellular telephone links, such as3G, LTE, or WiMAX, in some embodiments. Cell phone 66 and tabletcomputer 70 are portable computer systems that allow user 10 to utilizethe public speaking training system from any location with cellulartelephone service or Wi-Fi.

FIG. 4 illustrates cloud network 75. Cloud network 75 represents asystem of servers 80, applications 82, and remote storage 84 thatcomputer system 20 connects to and utilizes via communication link 32and electronic communication network 52. Computer system 20 utilizesfunctionality provided by servers 80, applications 82 served by orrunning on servers 80 or other servers, and remote storage 84 located atservers 80 or in other locations. Servers 80, apps 82, and storage 84are all used by user 10 connecting to a single uniform resource locator(URL), or using a single application on computer system 20, even thoughapps 82 and storage 84 may exist across a plurality of computer servers80. Computer system 20 connects to the various computer resources ofcloud network 75 transparently to user 10, as necessary to perform thefunctionality of the public speaking training program.

Cloud 75 is used in some embodiments to serve the program code for thepublic speaking training program to computer system 20 for use by user10 to practice a presentation. The training program exists as anapplication 82 in cloud 75 rather than on a mass storage device 24 localto computer system 20. User 10 visits a website for the training programby entering a URL into a web browser running on computer system 20.Computer system 20 sends a message requesting the program code for thetraining software from a server 80. Server 80 sends the application 82corresponding to the presentation training software back to computersystem 20 via electronic communication network 52 and communication link32. Computer system 20 executes the program code and displays visualelements of the application in the web browser being used by user 10.

In some embodiments, the program code for the public speaking trainingapplication is executed on server 80. Server 80 executes the application82 requested by user 10, and simply transmits any visual output tocomputer system 20. Computer system 20 streams the physical input datarepresenting a presentation by user 10, and any other data required forthe training program, to servers 80 via network 52. Servers 80 streamback a simulated environment and feedback to computer system 20.

Besides serving the presentation training program as an application 82,cloud 75 is also used to analyze the physical input data representing apresentation by user 10 in some embodiments. As user 10 gives apresentation to computer system 20, the computer system streamscollected data to servers 80 for analysis. Servers 80 execute programcode that analyzes the text of the presentation, as well as movement,eye contact, and other visual cues from video of the presentation toextract features, calculate metrics, and determine any feedback thatshould be given to user 10. Cloud 75 can be used to analyze thepresentation of user 10 whether the training program exists as anapplication 82 on cloud 75, or if the program code is installed andexecuted locally to computer system 20. In other embodiments, theprogram code running on computer system 20 performs all the analysis ofpresentation data locally to the computer system without transmittingthe presentation data to servers 80 on cloud 75.

A third use of cloud 75 is as remote storage and backup for presentationdata captured by the presentation training program. Computer system 20sends video, audio, and other data captured during a presentation byuser 10 to servers 80 which store the data in cloud storage 84 forfuture use. In some embodiments, video and audio of the entirepresentation is stored in storage 84 after the presentation. In otherembodiments, only the features, statistics, metrics, and other resultscalculated by computer system 20 or servers 80 based on the audio andvideo presentation data is stored in cloud storage 84. The presentationdata in storage 84 is used by user 10 at future times to review progresswithin the training program or to recall presentation tips and feedbackprovided by the training program.

Presentation data for a plurality of users can be aggregated withinstorage 84 for review by a manager or supervisor at a companyimplementing the training program across an entire employee base.Results for multiple users could also be reviewed by a professor at auniversity monitoring the progress of students. A manager logs into aprogram connected to cloud 75 to view aggregate presentation data foreach employee participating in the presentation training program. Themanagement program can be hosted on cloud 75 as an application 82. Themanagement program accesses the presentation data in storage 84 andpresents a dashboard to the manager. The dashboard shows eachparticipating employee and the progress being made. The manager canreview employee performance, see which employees are keeping up withlessons, and assess how well employees are progressing important skillsets. In embodiments where user 10 is simply an individual, and notparticipating in a corporate training program, result data can be storedon mass storage 24, locally to computer system 20, rather than onstorage 84 of cloud 75.

The presentation training program can be run totally on computer system20, or may be run completely on cloud 75 and simply be displayed oncomputer system 20. Any subset of the above described cloudfunctionality may be used in any combination in the various embodiments.In one embodiment, the functionality of the training application isimplemented completely on cloud 75, while in other embodiments thefunctionality runs completely on computer system 20. In someembodiments, the functionality is split between cloud 75 and computersystem 20 in any combination.

FIG. 5 illustrates presentation training application 100 including aplurality of software engines providing the functionality of theapplication. Application 100 can be stored on mass storage 24 as aninstalled program, stored in RAM 26 for execution, or stored in cloud 75for remote access. A software engine can be a library, a softwaredevelopment kit, or other object that denotes a block of softwarefunctionality. Software developers can purchase engines predesigned bythird parties to provide certain functionality of application 100, andthereby prevent having to completely rewrite the program code forfunctionality that has already been adequately implemented by others.Engines can also be written from scratch for the unique functionalityrequired to run application 100.

Application 100 includes graphics engine 102, human rendering engine104, physics engine 106, audio engine 108, speech analysis engine 110,and file input and output (I/O) engine 112. Other engines notillustrated are used in other embodiments to implement otherfunctionality of application 100. In some embodiments, a game enginesuch as the Unity engine, Torque3D, id Tech, or Unreal Engine is used.Game engines provide a graphics engine, physics engine, audio engine,networking engine, and other functionality common to most 3-D videogames in a single package.

Graphics engine 102 provides functionality to generate, render, anddisplay 3-D graphics. 3-D models for tables, chairs, and otherenvironmental objects, as well as the people in the crowd, are allloaded into graphics engine 102 by the program code of application 100.Application 100 communicates to graphics engine 102 the shape andlocation of each 3-D object to be rendered, and the graphics engineinterfaces with graphics hardware of computer system 20 to render anddisplay a 2-D image of the created scene. Animations are implemented byapplication 100 telling graphics engine 102 which 3-D objects to move,and how to move the objects. Graphics engine 102 helps the creators ofapplication 100 because interfacing with graphics hardware to generateand display a 2-D image from 3-D models is a challenging problem thathas been solved by many talented programmers before. There is no reasonfor a new graphics engine 102 to be created for presentation trainingapplication 100, so a third party graphics engine is purchased andintegrated into the presentation training application in mostembodiments.

Human rendering engine 104 is used to generate and animate 3-D models ofhuman beings. Instead of having to create new 3-D models of humanbeings, and program the models to animate properly, human renderingengine 104 is integrated into application 100 to provide 3-D models forthe crowd. A 3-D model of a person is generated by application 100simply by making an application programming interface (API) call tohuman rendering engine including arguments defining characteristics ofthe desired humans to draw. Human rendering engine 104 can create peopleof differing heights, weights, skin color, hair color, clothing, etc.,by simply varying the arguments to the engine API call. Human renderingengine 104 interfaces with graphics engine 102 to automatically renderand display generated 3-D models of human beings. A programmer creatingapplication 100 may not have the expertise to generate realistic 3-Dmodels of human beings and so purchases a third party engine to renderhuman models in some embodiments.

Application 100 uses human rendering engine 104 to generate a simulatedaudience corresponding to the size of the environment being simulated.For example, if an auditorium is selected for giving the presentation,application 100 uses human rendering engine 104 to generate a fewhundred audience members to fill the auditorium. If user 10 selects aboard room for the presentation, 5-10 audience members might begenerated. In some embodiments, audience members are customized based ona selection by user 10 of the type of audience. Upper management may berendered to be older and wearing suits, while students will be renderedas younger and having casual clothes. In some embodiments, audiencemembers are rendered with photo-realistic faces based on importedpictures of actual people who user 10 will be speaking to, e.g.,important executives at a company the user works at.

Physics engine 106 controls how rendered objects interact with eachother. Physics engine 106 approximates gravity so that loose objects inthe environment fall toward the ground. Physics engine 106 also causesobjects to bounce off of each other with realistic physics reactions.

Audio engine 108 interfaces with the sound hardware of computer system20. Audio engine 108 allows application 100 to capture audio from amicrophone connected to computer system 20 and play audio throughspeakers connected to computer system 20 without the programmer ofapplication 100 having to understand each underlying operating system orhardware call.

Speech analysis engine 110 receives the audio, video, biometric, andother data captured during a presentation by user 10, extracts featuresof the presentation from the data, and generates metrics, statistics,feedback, tips, and other output to help the user develop and improvepresentation skills. Speech analysis engine 110 is critical newfunctionality of presentation training application 100 and is programmedfrom scratch. However, in some embodiments, specific functionalityrequired to observe and extract features from a presentation by user 10is implemented using 3rd party software.

File I/O engine 112 allows application 100 to read and write data frommass storage 24, RAM 26, and storage 84 of cloud 75. File I/O engine 112allows the programmer creating application 100 to utilize various typesof storage, e.g., cloud storage, FTP servers, USB thumb drives, or harddrives, without having to understand each required command for each kindof storage.

Application 100 modularizes functionality into a plurality of softwareengines to simplify a programmer's task. Engines can be purchased fromthird parties where the functionality has already been created byothers. For functionality new to application 100, engines are createdfrom scratch. Each engine used includes an API that a programmer use tocontrol the functionality of the engine. An API is a plurality oflogical functions and data structures that represent the functionalityof an engine. The API of graphics engine 102 may include a function todraw a 3-D model on the screen, with a data structure representing the3-D model as an input to the API function. Graphics engine 102 alsoincludes functions to move a 3-D object, move the viewpoint the scene isrendered from, or remove a 3-D object from the screen. Audio engine 108includes an API function call to play a sound file through speakers ofcomputer system 20, or to read any cached audio information from themicrophone.

FIG. 6 illustrates a sample API 120 for human rendering engine 104. API120 includes new human function 122, delete human function 124, beginanimation 126, set state function 128, and list humans 130. Calls tofunctions 122-130 are integrated into the program code of application100 to render and animate a crowd on display 34.

A call to new human function 122 creates a new person and draws theperson on display 34. Any number of arguments to function 122 controlwhat the person looks like, the clothes the person is wearing, and wherethe person is located on the screen. The location of a new person can bespecified based on another object, e.g., by identifying a chair objectthe person should sit in, or the location may be specified by a 3-DCartesian coordinate indicating the location of the person in 3-D space.New human function 122 can be called once for each person desired to bedrawn in a crowd, or a single call can be made with arguments definingeach person to be drawn in the single call. Delete human function 124removes an already rendered human from the 3-D environment. An argumentto delete function 124 identifies which person in the crowd should bedeleted.

Begin animation function 126 causes a rendered human to execute ananimation. Arguments to begin animation function 126 indicate whichperson should do an animation, and define an animation that the renderedperson should perform. Human rendering engine 104 includes predefinedanimations that a programmer can easily make a person perform withouthaving to know a lot about 3-D animation. Human rendering engine 104includes built-in animations for motions such as head nodding, handwaving, talking, sitting down, and standing up. Other animations can begenerated using 3-D modeling software included with human renderingengine 104. A data structure defining the animation is stored along withthe program code of application 100, and given as an argument to beginanimation function 126 to execute the animation. In presentationtraining application 100, the animations executed are contingent inlarge part on the real-time performance of user 10 while using thetraining program.

Set state function 128 allows application 100 to set an emotional stateof a rendered human being, and human rendering engine 104 automaticallyanimates the person in a manner consistent with the emotional state. Aperson's emotional state could be set to agitated, in which case theperson would be animated as fidgety and nervous. Insofar as presentationtraining application 100, the crowd could be set to a state of engaged,neutral, or bored. Human rendering engine 104 automatically makes thecrowd models perform animations consistent with those states. Forinstance, an engaged crowd member is rendered to lean forward, nod, andmake direct eye contact in response to statements by user 10. A boredcrowd member may check the time, fall asleep, talk to a neighbor, orrest her head on her arm.

List humans function 130 returns a list of each human rendered by humanrendering engine 104. The returned list can be used by other portions ofapplication 100 that need to reference a specific crowd member. Listhumans function 130 is also used to perform a function on each renderedhuman at once, such as setting the state of an entire crowd to bored.

Human rendering engine 104 generates 3-D models of people as directed byapplication 100, and interfaces with graphics engine 102 to draw thehumans on a screen and animate the humans. Each engine 102-112 includesan API similar to API 120 used by a programmer to integrate thefunctionality of the engine into application 100.

FIG. 7 illustrates speech analysis engine 110. Speech analysis engine110 receives signals from the various input peripherals 28 of computersystem 20 and analyzes the inputs to extract features of thepresentation and generate metrics used by application 100 to observe andcritique a presentation given by user 10. The inputs to speech analysisengine 110 include microphone 140, camera 142, biometric reader 144, andpresentation materials 146. The audio from microphone 140 is routedthrough speech to text engine 148 to generate text of the words thatuser 10 is speaking. The text from speech to text engine 148 is routedas an additional input to speech analysis engine 110. Speech analysisengine 110 includes a vocalics analysis engine 150, a text analysisengine 152, a behavior analysis engine 154, a biometric analysis engine156, and a materials analysis engine 158.

Microphone 140 is electrically connected to a line-in or microphoneaudio jack of computer system 20. Microphone 140 converts analog audiosignals in the environment, e.g., speech from user 10, to an analogelectrical signal representative of the sounds. Audio hardware ofcomputer system 20 converts the analog electrical signal to a series ofdigital values which are then fed into vocalics analysis engine 150 ofspeech analysis engine 110. In other embodiments, microphone 140generates a digital signal that is input to computer system 20 via aUniversal Serial Bus (USB) or other port.

In one embodiment, microphone 140 is a part of a headset worn by user10. The headset includes both headphones for audio output by computersystem 20 to user 10, and microphone 140 attached to the headphones. Theheadset allows for noise cancellation by computer system 20, andimproves the audio quality for the presentation received by speech totext engine 148 and vocalics analysis engine 150.

Vocalics analysis engine 150 analyzes the sound generated by user 10,rather than the content of the words being spoken. By analyzing thesound from user 10, vocalics engine 150 identifies the pace at which theuser is speaking, how the pitch, volume, and pace of the user's voice ischanging, and the timing and length of pauses inserted by the user.Vocalics analysis engine 150 analyzes the rhythm, intonation, andintensity of the user's voice during a presentation. Vocalics analysisengine 150 provides an engagement score based on the amount ofvariability in select features of the voice of user 10. In oneembodiment, the engagement score provided by vocalics analysis engine150 is based on the pitch, pace, and volume with which user 10 speaks.

Speech-to-text engine 148 converts the audio signal of the voice of user10 into text representative of the words being spoken by the user. Thetext from speech to text engine 148 is provided as an input to speechtext analysis engine 152. Text analysis engine 152 analyzes the contentof the presentation by user 10. Text analysis engine 152 performsnatural language processing and determines linguistic complexity of thespeech, analyzes word choice, and marks the use of verbal distractors.

Verbal distractors are sounds or words such as “uhhh,” “ummm,”“basically,” and “like” which a speaker commonly uses to fill gaps ofsilence or while trying to remember what to say next. Linguisticcomplexity is an overall rating of the vocabulary being employed by user10. Text analysis engine 152 rates the linguistic complexity byeducation level. For example, user 10 may be rated as using words at amiddle school level, a university level, or at a professional level.Complexity is determined by performing syntactic analysis utilizinglanguage models.

Word choice analysis looks more specifically at the individual words andphrases used by user 10. Text analysis engine 152 flags a word thatappears to be used incorrectly by user 10, and also flags weak languagewhen another more effective word or phrase could be used. If user 10overuses a specific word or phrase, text analysis engine 152 may flaguses of the phrase to encourage the user to mix in a larger variety oflanguage. If user 10 knows she has a specific problem with certain wordsor phrases she doesn't want to say, the user can configure application100 so that speech text analysis engine 152 flags uses of theundesirable words. Flagged words and phrases are features output by textanalysis engine 152.

Speech text analysis engine 152 is programmed with specific words andphrases commonly used in specific domains of speech, e.g., in the techsector or among financial institutions. Speech text analysis engine 152generates a metric identifying how well user 10 is utilizing thelanguage of a specific domain where the user will be speaking, andsuggests word replacements to use language more fitting for the domain.Text analysis engine 152 uses linguistic analysis to generate metricsfor clarity and conciseness of speech, sentence structure, sentencelength, grammar, audience relatability, professionalism, and competency.Speech text analysis engine 152 extracts other features from the speechtext as user 10 presents. Speech text analysis engine 152 identifies afeature when user 10 begins a sentence, ends a sentence, begins anarrative, ends a narrative, etc.

Advanced analysis of the structure of a presentation is performed bytext analysis engine 152. Text analysis engine 152 analyzes thebeginning and ending of a speech to create a metric rating whether user10 properly opened and closed the speech, whether the main idea of thespeech has been clearly communicated, and whether the body of the speechis structured in a coherent manner. The text of the speech is analyzedto identify metaphors and contrasting language, and generate a metric ofproper metaphor use. Metaphors are also output as features. Storytellingor anecdotal elements in the text of the presentation are identified andoutput as features. A metric to gauge the amount and effectiveness ofstorytelling and anecdotes being used is also generated.

Text analysis engine 152 is able to identify emotional versus analyticalcontent of the presentation and generates a metric of the proportion ofanalytical and emotional content. A discourse clarity metric isgenerated that incorporates computational discourse analysis based onrhetorical structure theory and coherence models. The discourse claritymetric models the flow of concepts discussed in the presentation toidentify whether an audience member is likely to be able to follow theideas and logic of the presentation, and whether the sentences have asyntactic structure that is too complex for the intended audience of thepresentation.

Features and metrics output from vocalics analysis engine 150 arecombined with results from speech text analysis engine 152 to generate aperception metric. The perception metric rates or identifies how user 10is being perceived by a crowd. User 10 may be perceived by the crowd asenthusiastic, confident, charismatic, emotional, convincing, positive,competent, etc. The perception metric may include a numerical rating foreach possible perception category, e.g., a separate numerical indicatorof how enthusiastic, how confident, how emotional, how charismatic, andhow convincing user 10 is in her presentation.

Behavior analysis engine 154 receives a video stream of user 10performing a presentation. The video feed is received by application 100from camera 142 and routed to behavior analysis engine 154. Behavioranalysis engine 154 looks at the behavior of user 10 while presentingthe speech. Body movement, posture, gestures, facial expression, and eyecontact are all analyzed. Behavior analysis engine 154 looks at bodymovement, gestures, and posture of user 10 to flag or output a featureif the user is fidgeting her hands, rocking back and forth, orexhibiting other undesirable body movements while presenting. Behavioranalysis engine 154 observes the body of user 10 to ensure that the useris properly facing toward the audience. The body movement of user 10 isalso analyzed for proper use of hand gestures that match or complementthe text of the speech by tying together the outputs of speech textanalysis engine 152 and behavior analysis engine 154. Features areoutput by speech analysis engine 110 corresponding to hand gestures byuser 10. Body movement of user 10 is analyzed to ensure adequatemovement and gestures. Behavior analysis engine 154 generates a featurefor body movement, and flags if the user is too rigid in her appearanceor mechanical in her movements. In one embodiment, a third partysoftware application, e.g., Visage, or a hardware device, e.g., Tobii,is used to implement eye tracking. Third party software is also used totrack body movement in some embodiments.

Other peripheral devices may supplement the information received fromcamera 142. In one embodiment, user 10 wears wrist-bands or anotherperipheral that monitors the position of the user's hands relative toher body and reports hand movements to behavior analysis engine 154.Other motion capture methods are used in other embodiments. In someembodiments, two cameras 142 are used. Parallax between the two cameras142 helps give behavior analysis engine a depth of view and better gaugethe distance of each body part of user 10 from the cameras.

The facial expression of user 10 is monitored to generate a feature whenthe user does not maintain a desirable facial expression. User 10 shouldmaintain a happy and positive facial expression in most situations, butother facial expressions may be desirable when discussing a negativeopinion or relating a harrowing anecdote. Behavior analysis engine 154also helps monitor for nervous tics or other behavioral anomalies ofuser 10, such as randomly sticking out the tongue for no reason orblinking in an unsightly manner, by outputting those presentationfeatures to application 100.

Eye contact is monitored to ensure that user 10 sufficiently maintainsthe important connection with the audience that eye contact provides.The video of user 10 presenting is captured by camera 142, and behavioranalysis engine 154 analyzes the image to determine where the user islooking. Behavior analysis engine 154 determines how well user 10 ismaintaining eye contact with the crowd, and how well the user is movingeye contact across different areas of the crowd. The direction orlocation that user 10 is looking is output and stored as a presentationfeature.

Behavior analysis engine 154 creates a log of a presentation,identifying when user 10 is looking at the crowd, and when the user islooking elsewhere. Behavior analysis engine 154 outputs a feature whenthe eye contact state of user 10 changes, e.g., from looking down atnotes to looking at the crowd. Eye contact features of user 10 arecompared against pre-specified metric thresholds to generate an overalleye contact score. Statistics are available which identify whatpercentage of the time the user is looking at the crowd. The eye contactscore takes into consideration if user 10 looked at each person orsection of the room for approximately the same amount of time. If user10 exhibits a particular problem, such as staring down at her feet forlong periods of time, presentation training application 100 uses theinformation from behavior analysis engine 154 to identify the problemand provide tips and offer video lessons to the user to address theproblem.

In one embodiment, user 10 uploads presentation materials 146, such astext of a speech, presentation slides, or notecards to be used forreference during the presentation. User 10 toggles between viewing thepresentation materials and viewing the virtual audience. The amount oftime user 10 spends looking at presentation materials is considered byapplication 100 to be time not in eye contact with the audience, whilethe user is considered to have proper eye contact when the virtualaudience is shown instead of the presentation materials.

In some embodiments, a separate camera 142 is used zoomed in to capturea high quality image of the face of user 10. In embodiments with aseparate camera 142 for facial recognition, a first camera 142 is zoomedback to capture the entire body of user 10 and observe general bodymovement while a second camera 142 is zoomed in on the face of user 10to capture higher quality images for better facial recognition and eyecontact analysis. Object tracking can be used to keep the second cameratrained on the face of user 10 if the user moves around whilepresenting. In other embodiments, two cameras are zoomed out to captureuser 10 as a whole, in order to get a field of depth, and a third camerais trained on the face of the user.

Biometric reader 144 reads biometrics of user 10 and transmits a datafeed representing the biometrics to biometrics analysis engine 156.Biometrics analyzed by biometrics analysis engine 156 include bloodpressure, heart rate, sweat volume, temperature, breathing rate, etc.Biometric devices 144 are disposed on the body of user 10 to directlydetect biometrics, or are disposed at a distance and remotely detectbiometrics. In one embodiment, biometric reader 144 is an activitytracker that user 10 wears as a bracelet, watch, necklace, or piece ofclothing, that connects to computer system 20 via Bluetooth or Wi-Fi.The activity tracker detects heartbeat and other biometrics of user 10and transmits the data to computer system 20. In some embodiments,biometric reader 144 provides information as to movements of user 10which are routed to behavior analysis engine 154 to help the behavioranalysis engine analyze body movements of the user.

User 10 inputs her presentation materials 146, such as overhead slidesor handouts, to application 100 for analysis. Materials analysis engine158 looks at the materials 146 to provide metrics related to how welluser 10 is using slides. Metrics include a rating for the number ofpoints on each slide, the amount of time spent on each point, slidedesign, usage of text versus images, and the type and organization ofcontent. Presentation features extracted from presentation materials 146include when user 10 advances to the next slide, or when a new bulletpoint on the same slide is reached.

Each analysis engine 150-158 of speech analysis engine 110 outputsfeatures as user 10 performs a presentation. When a presentation featureis detected, such as a pause in speaking, usage of a certain word, or abreak in eye contact, a result signal is generated by a respectiveanalysis engine 150-158. Application 100 captures the features andperforms further analysis to determine overall scores and ratings of theperformance, generate tips and suggestions, and provide real-timefeedback. Application 100 captures the results and outputs of analysisengines 150-158, and analyzes the results based on predetermined metricsand thresholds.

To interpret the features and metrics from speech analysis engine 110, asupervised machine classification algorithm is used, as illustrated inFIG. 8. Prerecorded speeches 170 are input into speech analysis engine110 to extract features and generate metrics for each of the prerecordedspeeches. The features and metrics from speech analysis engine 110, aswell as ratings 172 provided by experts who have observed the speeches170, are input into machine learning algorithm 174. Machine learningalgorithm 174 is used to generate a predictive model 176. Predictivemodel 176 defines correlations between features and metrics from speechanalysis engine 110 and ratings 172 of speeches 170 provided by publicspeaking experts.

Thousands of speeches 170 are input into speech analysis engine 110 toform the basis of predictive model 176. A wide variety of speeches, bothgood and bad, are input into the machine learning algorithm. Each speechis input into speech analysis engine 110 to generate the same featuresand metrics that will be generated when user 10 uses presentationtraining application 100. In addition, experts are employed to observespeeches 170 and provide ratings 172 based on the experts' individualopinions. In one embodiment, six public speaking experts rate eachindividual speech 170 to provide the expert ratings 172.

Machine learning algorithm 174 receives the features and metrics fromspeech analysis engine 110, as well as the expert ratings 172, for eachspeech 170. Machine learning algorithm 174 compares the key features andmetrics of each speech 170 to the ratings 172 for each speech, andoutputs predictive model 176. Predictive model 176 includes ratingscales for individual metric parameters and features used by application100 to provide ratings to a presentation subsequently given by user 10.Predictive model 176 defines what features make a great speech great,and what features occur that result in a poor expert rating.

Presentations of user 10 are compared against predictive model 176 toprovide tips and feedback. Prior to doing a presentation for analysis byapplication 100, user 10 performs an initial setup and calibration asshown in FIG. 9. FIG. 9 shows computer window or screen 180 with setupand calibration options 182-190.

Gender option 182 allows user 10 to enter a gender. An accurate gendersetting helps speech analysis engine 110 interpret data from microphone140 and camera 142. Skill level option 184 tells application 100 anapproximate starting level for the presentation skills of user 10.Setting skill level option 184 accurately helps application 100 presentlessons and topics that are most relevant and helpful to user 10. Abeginner will get more remedial lessons. An expert speaker will getadvanced lessons, or will be presented with exercises to maintain skillsrather than lessons to teach skills.

Options 186-190 take user 10 to other screens where calibration occurs.Calibrate speech recognition option 186 takes user 10 to a screen thatwalks the user through a calibration process to learn the voice andspeaking mannerisms of the user. User 10 is prompted to speak certainwords, phrases, and sentences. The calibration process analyzes how user10 speaks, and uses the data to interpret subsequent presentations usingspeech-to-text engine 148. Proper calibration helps application 100generate an accurate textual representation of a presentation by user10, which improves analysis accuracy of the content of the presentation.

Calibrate eye tracking 188 takes user 10 to a screen where application100 is calibrated to better recognize where exactly the user is looking.User 10 is asked to move to various locations in the room, and look atdirections dictated by application 100. Application 100 analyzes theface of user 10 from various angles and with eyes looking in variousdirections, and saves a model of the user's face for use in determiningwhere the user is looking during a presentation. In one embodiment, theeye tracking calibration routine displays a dot that moves arounddisplay 34 while the eye calibration routine accesses webcam 142 toobserve the eye movement and position of user 10 following the dot.

Calibrate facial recognition 190 is used to learn the features of theface of user 10. Photos of the face of user 10 are taken with webcam 142from various angles, and the user is also prompted to make variousfacial expressions for analysis. User 10 may also be asked to confirmthe exact location of facial features on a picture of her face. Forinstance, user 10 may be asked to touch the tip of her nose and thecorners of her mouth on a touchscreen to confirm the facial recognitionanalysis. Facial recognition calibration helps speech analysis engine110 accurately determine the emotions being expressed by user 10 whilepresenting. In one embodiment, facial recognition of presentationtraining application 100 is fully automatic, and no calibration isrequired to track mouth, chin, eyes, and other facial features. In otherembodiments, calibration is not required, but may be used for enhancedprecision.

In one embodiment, after setup and calibration is completed using page180, application 100 uploads the configuration data to storage 84 ofcloud 75. Uploading configuration data to cloud storage 84 allows user10 to log into other computer systems and have all the calibration dataimported for accurate analysis. User 10 can configure application 100 ona home personal computer, and then practice a presentation in a hotelroom using a mobile telephone. The mobile telephone is automaticallysetup and calibrated to the user's voice and face by downloadingconfiguration data from cloud storage 84. In some embodiments, a portionof the calibration is required to be performed again if a new type ofdevice is used, or when a different size of screen is used.

FIG. 10 shows a screen 200 used by user 10 to begin a presentationtraining session using application 100. User 10 can do exercises orlessons using option 202, guided practice with option 204, self-practicewith option 206, or review the analysis of past performances with option208.

Lessons button 202 includes a summary of progress that user 10 hascompleted to date, e.g., so far user 10 has completed 4 of the 19 totallessons. Clicking or touching lessons button 202 takes user 10 toanother screen with a more thorough explanation of each available lessonand any progress already completed. Examples of lessons that areavailable include lessons on enunciation and pronunciation of words,storytelling, improving eye contact, and other best practices for publicspeaking. Each lesson may include an indication of whether the lessonhas been completed, and how well user 10 performed on the lesson.

The lessons of application 100 include instructional videos put togetherby professional instructors and interactive elements to guide the userthrough learning specific skills critical to public speaking. Afterreceiving instruction on a skill, and instruction on an activity thatthe user performs to exercise the skill, the user responds to promptsand performs short bits of speaking to demonstrate an understanding ofand ability to perform the skill.

User 10 progresses through the lessons one by one. User 10 may work on asingle lesson until the skill is perfected, or may do each lesson oncethen go back to perfect select lessons. User 10 may decide to only doselect lessons that the user knows she needs to work on. Application 100may suggest after user 10 practices a speech that the user return tospecific lessons to remediate certain skills. Progress through lessonsis uploaded to cloud storage 84 for review by user 10 on other devicesor by a supervisor of the user in a corporate or academic environment.

User 10 does guided practice by clicking or touching button 204. Inguided practice, application 100 generates a hypothetical scenario foruser 10 to practice a presentation. Application 100 gives user 10 asample topic to speak on, or gives prompts for the user to answer. User10 responds to the prompts, or speaks on the given topic for the allowedamount of time, and then application 100 rates the presentation andgives feedback.

Self-practice is performed by clicking or pressing self-practice button206. Self-practice allows user 10 to practice any speech on any topicthat the user needs to present. In one embodiment, after pressingself-practice button 206, user 10 is asked to enter information aboutthe presentation. Entering information such as desired length ofpresentation, topic of presentation, and technical expertise of crowd,helps application 100 perform analysis tailored to the particular typeof presentation user 10 will be giving, and the type of audience user 10will be speaking in front of. Application 100 can make sure that user 10uses technical language appropriate for the technical level of theaudience, and uses proper technical terms for the field of expertise.

User 10 also configures the room and the crowd to approximate the typeof presentation the user will be giving. For instance, if user 10 isspeaking before seven board members in a boardroom, application 100 willsimulate that environment for the user's presentation practice. If user10 will be speaking in front of thousands of people in an auditorium,then application 100 renders a large auditorium and fills the auditoriumwith thousands of rendered humans. Application 100 further simulatesdifferent environments by adding audio effects. Application 100simulates the sound and echo of large auditoriums and stadiums wherelarge arrays of speakers will likely be used. Application 100 likewisesimulates the sound of other environments, including adding applause orother crowd noises.

Review performance button 208 allows user 10 to review each pastpractice performance to see what went right and what went wrong, reviewtips and feedback, or watch a performance in whole. Both guided practiceand self-practice can be reviewed. Lessons are reviewed under lessonbutton 202, but in some embodiments can also be reviewed under reviewperformance button 208. In addition to analysis and recordings of eachpast presentation user 10 has completed, application 100 presentssummaries of performance trends over time. If user 10 has been steadilyimproving certain skills while other skills have stayed steady orworsened, the user will be able to see those trends under reviewperformance button 208.

User 10 selects a presentation mode from screen 200, and then beginsdoing a practice presentation. FIG. 11 illustrates the process ofapplication 100 analyzing a presentation by user 10. Physical userinputs 220 from input peripherals 28 are provided to speech analysisengine 110. Speech analysis engine 110 interprets physical user inputs220 with the aid of the calibration and setup 222 that the userpreviously performed. Speech analysis engine 110 outputs identifiedfeatures, calculated metrics, and other information that application 100interprets through predictive model 176 to generate real-time feedback224 and scores and ratings 226.

Physical user inputs 220 include microphone 140, camera 142, andbiometric reader 144. User 10 also provides any presentation materials146 being used if available. Speech analysis engine 110 receives thephysical data generated by user 10 giving a presentation, and analyzesthe content of the speech as well as the way the speech is beingperformed. Calibration 222 helps speech analysis engine 110 analyzephysical inputs 220 because the speech analysis engine becomes aware ofcertain idiosyncrasies in the way user 10 pronounces certain words, orthe way the user smiles or expresses other emotions through facialexpressions.

Speech analysis engine 110 extracts features and generates metrics inreal-time as user 10 performs a presentation. The features and metricsare all recorded for future analysis, but are also routed to predictivemodel 176 for comparison against various thresholds contained within thepredictive model. Based on how the presentation by user 10 compares tothe speeches 170 that were expertly rated, application 100 generatesreal-time feedback during the presentation and scores and ratings forpresentation after the presentation is complete.

Real-time feedback comes in the form of the 3-D simulated crowd andenvironment, as well as alerts and notifications. Application 100renders a simulated crowd to watch the presentation of user 10. Therendered crowd reacts realistically to the presentation. If user 10 isengaging, the crowd will be rendered as attentive and observant, but ifthe user is not engaging, the audience is rendered as being bored andnonobservant. Application 100 also provides optional audible andon-screen alerts and status updates. Application 100 may display a graphof certain metrics over time that user 10 wants to keep an eye on duringthe presentation. An audible ding may be used every time user 10 uses averbal distractor to train the user not to use distractors. Visible andaudible real-time feedback is configurable, and application 100 includesan option to completely disable real-time feedback. User 10 presentsuninterrupted and reviews all feedback after the presentation.

Scores and ratings 226 are provided by application 100 when user 10completes a presentation. Scores and ratings 226 reflect the featuresand metrics of an entire presentation, and may be based on peaks,averages, or ranges of metric values. Multiple scores are provided whichare each based on a different combination of the metrics and featuresgenerated by speech analysis engine 110. In one embodiment, one overallscore is presented, which combines all of the presentation attributes.

FIGS. 12 a-12 e illustrate user 10 giving a simulated presentation usingapplication 100. In the embodiment of FIGS. 12 a-12 d, a largeflat-screen television is used for the presentation. In the embodimentof FIG. 12 f, a virtual reality headset is used. In other embodiments,other types of screens, such as projectors, phones, tablets, or computermonitors, are used.

FIG. 12 a illustrates user 10 giving a simulated presentation in frontof television 240. A physical input device or sensor array 242 ismounted over television 240. Sensor array 242 includes microphone 140,camera 142, and biometric reader 144. A computer 244 is set adjacent totelevision 240 to run application 100. Application 100 renders a virtualaudience including audience members 252, 254, and 256 on the screen oftelevision 240. Microphone 140 receives sound waves from user 10speaking during the presentation and converts the sound waves to anelectrical signal. Microphone 140 is connected to computer 244 so thatapplication 100 is able to receive input from the microphone. In someembodiments, an analog signal is generated by microphone 140 andconverted to a digital signal by sound hardware of computer 244. Inother embodiments, microphone 140 includes a built-in analog-to-digitalconverter and transmits a digital signal to computer 244.

Camera 142 transmits an image of user 10 to computer 244, so thatapplication 100 can observe movement of the user during a presentation.Biometric sensor 144 is a thermographic camera in the embodiment of FIG.12 a. Other types of biometric sensors are used in other embodiments.Thermographic camera 144 is sensitive to infrared radiation and createsa thermal image of user 10. The thermal image is useful to gaugetemperature, heartbeat, and breathing patterns of user 10. User 10 givesa presentation in front of television 240 and sensor array 242.Application 100, running on computer 244, generates a simulatedenvironment and a simulated crowd, and then analyzes the presentation.

The simulated audience includes realistic renderings of human beings,and the people in the audience react realistically to the presentationby user 10. If application 100 determines that user 10 is not beingengaging, the application renders the audience in a manner indicatingboredom. For instance, in FIG. 12 b, audience member 252 is looking downat her lap, audience member 254 is checking the time, and audiencemember 256 is staring out of a window at the side of the room. Theaudience members 252-256 are animated, and perform different actionsover time to indicate boredom. An audience member may check the time atone point, then put his or her head down on the table a little bitafterwards. Other actions indicating boredom include looking anxious toleave, talking amongst each other in the crowd, or head scratching.

On the other hand, if user 10 is engaging in her presentation, theaudience members are rendered as being attentive and interested. In FIG.12 c, audience members 252 and 254 are rendered as leaning forward intheir seats. Audience member 256 is nodding his head in agreement with apoint user 10 is making. Audience members may also laugh at an anecdote,clap at appropriate times, and make good eye contact to indicateinterest or engagement. In one embodiment, the audience may be in threedifferent states: engaged, disengaged, or neutral. In other embodiments,more levels of engagement are used.

FIG. 12 d illustrates examples of real-time feedback 224. Application100 displays a feature or metric graph 260 while user 10 is presenting.User 10 configures metric graph 260 to display metrics that the user ishaving trouble with, or wants to practice. In other embodiments,application 100 displays any metric or feature that the applicationdetermines is of importance to user 10 at a particular moment. Themetric or feature graph is rendered to change over time as thepresentation by user 10 progresses. Values for the features and metricsare recalculated periodically and graph 260 is updated to show how thevalues change. The metric graph 260 may grow to the right as timepasses, zoom out over time to stay the same size but still show a graphof the entire presentation, or only show the most recent period of time,e.g., the last thirty seconds of the presentation.

Alert or notification 262 indicates when a metric is outside of athreshold goal. In FIG. 12 d, user 10 is going faster than a goal set byapplication 100 for the user. Application 100 uses notification 262 toalert user 10 to pay attention to the pace of the presentation and slowdown a bit to meet the goal. Other notifications are possible for anyreason that application 100 may find use in communicating with user 10during a presentation. Notifications 262 are used to alert user 10 whentime for the presentation is running out, when the user is moving aroundtoo much, or when the user needs to smile more.

Counter 264 is used to keep user 10 notified of the number of verbaldistractors being used. User 10 may configure application 100 to showthe distractor counter because the user knows a problem exists.Application 100 may also automatically show distractor counter 264 iftoo many distractors are used. Counter 264 may be used to show the totalof all distractors used, or one particular distractor that user 10 usesexcessively. Other counters are used in other embodiments. In oneembodiment, a running total of the number of anecdotes is shown, or atimer showing elapsed or remaining time is displayed.

FIG. 12 e illustrates user 10 using a virtual reality (VR) headset 266for application 100. Audience members 252-256 are projected in 3-D onsmall display screens within VR headset 266. The audience is drawn as adashed line in FIG. 12 e to indicate that the audience does not exist inreal life, but that to user 10 the audience appears at the locationindicated. VR headset 266 understands the orientation of the head ofuser 10 and changes the display so that when the user moves, theaudience always appears to be at the same location relative to the roomthe user is in. If the user 10 looks down, the audience is displayedfurther up within VR headset 266. If the user 10 looks up, the audienceis displayed lower in VR headset 266. Application 100 is able tocalculate the direction 268 that user 10 is looking. In FIG. 12 e, lineof sight indicator 268 shows that user 10 is making eye contact withaudience member 254.

In some embodiments, sensor array 242 is used along with VR headset 266to gather additional information about the movement and orientation ofthe body parts of user 10. VR headset 266 may include cameras trained onthe eyes of user 10 to determine where the user is looking with higheraccuracy. VR headset 266 may be a mobile device that operates separatelyfrom any other computer system and connects to cloud 75 for somefunctionality. Samsung Gear VR, Oculus Rift, or Google Cardboard devicesare used in some embodiments.

In other embodiments, augmented reality devices, such as MicrosoftHoloLens, are used instead of virtual reality devices. Augmented realitydevices allow user 10 to see the environment around her, but are able tooverlay simulated audience members into the real environment. User 10 isable to give a practice presentation in the actual room where a realperformance will later be given, and application 100 simulates arealistic audience in the room.

In one embodiment, no display 240 is used while presenting. User 10wears a headset with headphones and microphone 140 while presenting.Training application 100 receives and analyzes an audio signal of thepresentation from microphone 140 without presenting real-time feedbackusing display 240.

FIG. 13 illustrates a summary page that is displayed after user 10completes a presentation. Application 100 gives the presentation anoverall score 270 in a letter grade or numerical form, and providesadditional metrics, statistics, and scores. Application 100 reportstotal presentation time 272, and can also notify user 10 how the totaltime of the presentation compared to a goal set by the user or by theapplication. Average engagement 274 is presented as a total out of 100,but may also be a letter grade or other score. Average pace 276 isdisplayed, and is compared to a goal that user 10 is trying to meet. Thelevel of linguistic complexity used is displayed at 278. The level offidgeting during the presentation is displayed at 280, because user 10has displayed excessive fidgeting in the past. In the presentation justcompleted in FIG. 13, user 10 has met the goal of reducing fidgeting toa minimum. Eye contact has been graded a B by application 100, and isdisplayed at 282. The posture of user 10 has been graded an A−, and isdisplayed at 284.

Application 100 shows user 10 a timeline 286 of the presentation.Timeline 286 represents the entire presentation from beginning to end,and includes periodic vertical time markers to help orient user 10.Points of interest 288 are displayed on the timeline as exclamationpoints, stars, or other symbols, and show the user where good or badevents happened during the presentation. In one embodiment, a firstsymbol is used to mark where the user performed especially well, and adifferent symbol is used to mark where the user did something that needscorrection.

User 10 clicks or touches one of the points of interest 288 to pull up ascreen with additional information. A popup tells user 10 what wentright or what went wrong at that point of the presentation. The popupalso links to any lessons or tutorials that might be helpful incorrecting a problem. A video window allows user 10 to view hispresentation beginning right before the point where something ofinterest occurred. User 10 clicks through all of the points of interest288 to see each aspect of the presentation that application 100determined needs attention, and continues taking lessons or practicingto get better at public speaking.

FIG. 14 illustrates a screen 300 for user 10 to review a video 302 of apublic speaking performance. Screen 300 is accessed by clicking timeline286 or a point of interest 288. Clicking timeline 286 plays thepresentation in its entirety while clicking a point of interest 288plays only the pertinent portion of the presentation video.

Pace box 304 presents information regarding the pace with which user 10spoke over time. Graph 305 shows a pace rating change over time alongwith video 302. An overall pace rating 306 shows that overall user 10spoke with an excellent pace. Analysis button 308 is clicked by user 10to view detailed analysis regarding speaking pace including specifictips and tutorial videos.

Pauses box 310 presents information regarding pauses in speaking takenby user 10 during the presentation. Graph 312 illustrates when pausesoccur over time along with video 302. The specific time currentlyplaying in video 302 is illustrated at the dotted line at the center ofgraph 312. User 10 can see upcoming pauses approaching from the rightside of graph 312. Overall pauses rating 314 shows that user 10 did“good” with pauses during the presentation. Analysis button 316 opens apopup or another page explaining how the “good” rating was calculatedand showing tips and tutorial videos for improving usage of pauses byuser 10.

Pitch variability box 320 presents information regarding the use ofpitch by user 10 during the presentation. Graph 322 shows a rating ofpitch variability over time and follows along with video 302. Overallpitch variability rating 324 shows that user 10 did “average” for thepresentation as a whole. Analysis button 326 opens a popup or anotherpage showing detailed analysis of pitch variability by user 10.Engagement box 330 shows that user 10 did “good” with audienceengagement over the presentation as a whole. Graph 332 may highlightwhen specific events affecting engagement occur in video 302. Analysisbutton 336 shows a complete analysis of how the “good” rating ofengagement was determined.

FIG. 15 a illustrates screen 340 with a report viewed by a supervisor ata company to monitor performance of employees. Screen 340 could be usedin other situations, such as a professor or teacher monitoring theperformance of students. Summary 342 shows information regarding thecompany or university and statistics about the employees or students asa whole. Summary 342 shows the company name, average score of employees,total number of employees, and the total practice time of the employees.Other pertinent information is presented in other embodiments.

A table of users 344 is presented below summary 342. The users 344 areemployees, students, or other set of users being monitored by asupervisor, manager, teacher, etc. Location column 346 shows thelocation of each user 344. Sessions column 348 shows the number oftraining sessions that each user has participated in. Total time column350 shows the amount of presentation time that each user hasparticipated in. In some embodiments, the table shows total time ortotal sessions over all time, and over the last month. Last sessioncolumn 352 shows the date of the last training session performed by eachuser. Average score column 354 shows the average score over all sessionfor each user. In other embodiments, the average score column 354 iscalculated over a certain number of presentations or over a limitedperiod of time. View details column 356 includes buttons clickable by amanager to view additional information regarding individual users.

FIG. 15 b illustrates a screen 360 displayed to a manager after clickinga view details button 356 from screen 340 of FIG. 15 a. Summary section362 displays the name and other personal information of the specificuser the manager clicked on. Summary section 362 further displaysperformance information such as total number of sessions, averagepractice time per session, total practice time across all sessions, andthe date of most recent practice session. Other information is presentedin other embodiments. Table 364 illustrates a performance rating for themost recent 10 practice sessions of the user. The manager can click aparticular session from 364 to get detailed information and review videoof the session on a screen similar to screen 300 in FIG. 14. In oneembodiment, table 364 is scrollable so that information about all pastpresentations is viewable.

The presentation training application 100 utilizes a 3-D simulation,speech recognition, vocalic analysis, facial recognition, othernon-verbal feature analysis, and video hints to provide individualizeddynamic feedback to user 10. A practice presentation by user 10 iscompared against hundreds or thousands of sample speeches which havebeen expertly rated. A rating scale is obtained by rating each of thesample speeches and comparing the ratings to the parameters of thesample speeches, such as pace, pitch variability, volume, volumevariability, pauses, types of pauses, complexity of language, verbaldistractors, eye contact, facial gestures, body movement, andpresentation materials. A 3-D rendered audience is simulated in anenvironment and reacts in real-time. An overall performance score isassigned when the presentation is complete based on the individualparameters. Application 100 can be used as an on-demand training andpractice environment for public speaking and presentation or as alearning object in a communications course. Application 100 includes aseries of lessons introducing best practices through short videos, andallowing user 10 to practice individual concepts. Application 100improves on prior art presentation training programs by providingreal-time feedback, expert ratings, and gamification elements tomaintain engagement.

User 10 is awarded badges, points, and ranks based on the parameterssuch as practice time, score, improvement of metrics over time, etc.Users can log into an online dashboard to view their badges and ranks,compare earned badges to other students or employees, and see whatbenchmarks need to be reached to earn new badges or advance to the nextrank. The gamification features improve engagement between user 10 andapplication 100. User 10 uses application 100 on a continuing basis toimprove and maintain public speaking skills.

While one or more embodiments of the present invention have beenillustrated in detail, the skilled artisan will appreciate thatmodifications and adaptations to those embodiments may be made withoutdeparting from the scope of the present invention as set forth in thefollowing claims.

What is claimed is:
 1. A method of public speaking training, comprising:providing a computer system including a display monitor; providing amicrophone coupled to the computer system; providing a video capturedevice coupled to the computer system; rendering a simulated environmentincluding a simulated audience member on the display monitor using thecomputer system; recording a presentation onto the computer system usingthe microphone and video capture device; extracting a first feature ofthe presentation based on data from the microphone and video capturedevice while recording the presentation; calculating a metric based onthe first feature; animating the simulated audience member in responseto a change in the metric; generating a score based on the metric; anddisplaying the score on the display monitor of the computer system afterrecording the presentation.
 2. The method of claim 1, further including:providing a biometric device coupled to the computer system; andextracting a second feature of the presentation based on a data from thebiometric device.
 3. The method of claim 1, further including displayinga training video on the display monitor.
 4. The method of claim 3,further including suggesting the training video based on the metric orscore.
 5. The method of claim 1, further including drawing anotification on the display monitor based on the metric while recordingthe presentation.
 6. The method of claim 1, wherein providing thecomputer system includes providing a virtual reality headset.
 7. Amethod of public speaking training, comprising: providing a computersystem including a display monitor; providing a microphone coupled tothe computer system; providing a video capture device coupled to thecomputer system; rendering a simulated audience member on the displaymonitor using the computer system; recording a presentation onto thecomputer system using the microphone and video capture device;extracting a feature of the presentation based on a data from themicrophone and video capture device while recording the presentation;and animating the simulated audience member in response to the feature.8. The method of claim 7, further including providing a topic as aprompt for the presentation.
 9. The method of claim 7, further includingproviding a second video capture device coupled to the computer system.10. The method of claim 7, wherein providing the computer systemincludes providing a video game console.
 11. The method of claim 7,further including storing the feature on a cloud storage.
 12. The methodof claim 7, further including calculating a metric based on the featureusing a supervised machine learning classification algorithm.
 13. Themethod of claim 7, further including rendering the simulated audiencemember using a video game engine.
 14. A method of public speakingtraining, comprising: providing a computer system including a displaymonitor; providing a microphone coupled to the computer system;rendering a simulated audience member on the display monitor using thecomputer system; recording a presentation onto the computer system usingthe microphone; and animating the simulated audience member based on adata from the microphone.
 15. The method of claim 14, further includingproviding a biometric device coupled to the computer system.
 16. Themethod of claim 14, further including suggesting a training video basedon the presentation.
 17. The method of claim 14, further includingrendering the simulated audience member using a video game engine. 18.The method of claim 14, further including displaying a notification onthe display monitor while recording the presentation.
 19. The method ofclaim 14, further including providing a topic as a prompt for thepresentation.
 20. The method of claim 14, wherein providing the computersystem includes providing a tablet computer or mobile phone.
 21. Amethod of public speaking training, comprising: providing a simulatedaudience member; receiving a presentation; and animating the simulatedaudience member in response to the presentation.
 22. The method of claim21, further including animating the simulated audience member to appearbored or engaged.
 23. The method of claim 21, further including:extracting a feature of the presentation; and animating the simulatedaudience member based on the feature.
 24. The method of claim 21,further including providing the simulated audience member using avirtual reality headset.
 25. The method of claim 21, further includingproviding real-time feedback based on the presentation.